Handling a Global Software Failure: A Case Study

During a global software failure, I led the resolution efforts across multiple regions, including Japan, Asia, EMEA, LATAM, and North America. The issue was initially reported by a user in Japan and quickly spread to other regions. Coordinating with network engineers and resellers, I gathered detailed information from users and kept stakeholders informed throughout the process. By developing PowerShell scripts and SOPs for a complex uninstall process, and leveraging VP support to navigate legal restrictions, I ensured minimal downtime and restored normal operations efficiently.
WeWork
Programme Management
Operations
Case Study
Author

Eimhin Rafferty

Published

May 22, 2024

Modified

May 22, 2024

Week 1 Breakdown

Day 1: Initial Investigation

  • Japan Issue Reported: A user in Japan reports a software failure via our Jira ticketing platform.
  • Initial Investigation: Reviewed the report, checked service status pages, and confirmed no provider outages.
  • EMEA Impact: European and North American teams began reporting the same issue. Contacted network engineers suspecting a recent license renewal may be the cause.
  • Communication Loop: Informed managers and VPs about the issue and ongoing efforts to triage it.

Day 2: Advanced Triage

  • Japan Team Meeting: Learned from the Japan team that a deep uninstall process was suggested. Advised them to wait for my review.
  • Information Collection: Collected detailed information from affected users using Jira, providing templates for specific data.
  • Advanced Triage: Further calls with technical support and resellers did not find a resolution.
  • Communication Loop: Continued to inform leadership about the progress.

Day 3: Identifying Key Factors

  • Unaffected Group: Identified a franchise team using network licenses but managing their own hardware was unaffected.
  • Emergency Calls: Scheduled calls with key IT leaders to discuss potential causes.
  • VP Leverage: Utilized VP support to get key leaders on calls.
  • Stakeholder Updates: Kept all stakeholders informed about findings and ongoing investigations.

Day 4: Escalating the Issue

  • Service Blockage: The outage began severely impacting teams’ work.
  • Running out of Options: Discussions revealed deep reinstall as the only viable solution.
  • Litigation Hold: Legal restrictions prevented software uninstallation. Leveraged VP to push for a resolution.
  • Communication Loop: Broadened information loop to sales and operations teams to alleviate pressure on effected users.

Week-by-Week Breakdown

Week 2: Compliance Resolution

  • Agreement with Vendor: Reached an agreement to use a manual process in conjunction with their compliance app to collect logs directly from users.
  • Initial Workflow Test: Worked with team members on critical projects to test the workflow for log collection, and run the process end to end.

Week 3: Automation and Initial Resolutions

  • Automation Scripts: Developed PowerShell scripts to automate compliance checks and configured Jira for progress tracking.
  • First Reports Sent: Submitted the first batch of reports to the vendor, applied pressure for quick responses.

Week 4: Final Resolution

  • User Instructions: Provided users with detailed SOPs and scripts for the deep clean process, tracked progress via Jira.
  • Resolution Achieved: Despite slow vendor responses, the issue was resolved. Regular updates ensured transparency and maintained trust.
  • Close Out: Automated reports from Jira to monitor ticket burn down rates.

Conclusion

By systematically addressing the software failure, leveraging automation, and ensuring constant communication, I minimized downtime and restored normal operations, demonstrating effective problem-solving and leadership in crisis management.

Archilogic Editor 4.0: A Comprehensive Rebuild

As the product manager in Archilogic, I oversaw the rebuild of Archilogic Editor 4.0, introducing a Spacegraph data structure for floor plans. This project enhanced efficiency and accuracy, improved data security, and streamlined workflows.
May 29, 2024
Eimhin Rafferty

Automating Space Calculation: A Case Study on Leading Innovation in Real Estate with Archilogic

As a Product Manager at Archilogic, I led the development of a groundbreaking product that automates space calculations based on BOMA standards. This product leverages the power of our Spacegraph data structure, making it scalable, repeatable, and uniquely effective due to the relational nature of the underlying data.
May 21, 2024
Eimhin Rafferty

Case Study: Automating Floor Plan Polylining and Integration with IWMS Platforms

As a Product Manager at Archilogic, I led the development and integration of a solution to automate floor plan updates for customers, addressing a major pain point: keeping floor plans updated across multiple platforms, particularly their Integrated Workplace Management Systems (IWMS). This case study details how we collaborated with customers using FM Systems, developed automation for polyline creation, integrated our solution with IWMS platforms, and ultimately built a native app plugin for seamless data transfer.
May 27, 2024
Eimhin Rafferty

Case Study: Enabling Multiple Layouts at Archilogic

In Archilogic, I led a transformative project called ‘Enable Multiple Layouts.’ This initiative aimed to allow users to create, review, and implement different layout options for building floors without disrupting the main production environment. The project involved significant technical complexity, including new API implementations and enhancements to our space graph data structure.
May 20, 2024
Eimhin Rafferty

Case Study: Implementing Design Standards for EMEA - A Design Technology Product Management Initiative at WeWork

As a Design Technology Product Manager at WeWork, I led a high-impact project to translate global design standards into actionable content, processes, and training systems for teams in EMEA. This initiative was configured to address local needs, supply chain demands, and building regulations, ensuring consistency and compliance across the region.
May 19, 2024
Eimhin Rafferty

Case Study: Transforming Design Technology Support at WeWork

As a Design Technology Product Manager at WeWork, I spearheaded a critical initiative to overhaul the support process for our design technology teams. Recognizing the need for a systematic approach to support as our teams grew leaner and more remote, I led the design, implementation, and operationalization of a comprehensive Jira-based ticketing system. This project highlighted themes of empathy, accountability, accuracy, and cross-functional leadership, ensuring fair and transparent support for all teams.
May 26, 2024
Eimhin Rafferty

Developing a Workflow Template for the Custom Enterprise Team

At WeWork, I was tasked with supporting a newly formed ‘Custom Enterprise’ team within the Sales Organisation. This team of six designers was responsible for rapidly creating sales collateral for various buildings in our portfolio. Assembled on an ad-hoc basis and not reporting to the main design organisation, they faced significant operational challenges.
May 23, 2024
Eimhin Rafferty

Protecting 3.7 TB of Project Data: The BIM 360 Data Migration Initiative

The BIM 360 Data Migration project addresses the challenge of managing 3.7 TB of project data stored in a locked in cloud platform - ‘BIM 360 Teams’ which became inaccessible as versions of Autodesk Revit 2017/2018 were phased out. This migration ensureed that all project data moved to the sucessor, ‘BIM 360 Docs’, upgrading models to the latest Revit versions and maintaining user access throughout the process.
May 28, 2024
Eimhin Rafferty

WeWork Self Serve Platform: Enhancing Access and Security

To address significant bottlenecks and security issues, I led the design and development of a self-serve web app integrated with Autodesk Construction Cloud. This platform revolutionized how our global design team accessed and managed 3D models of WeWork’s 550 buildings and 2500 floors, ensuring both security and efficiency.
May 25, 2024
Eimhin Rafferty
No matching items